<!DOCTYPE html SYSTEM "about:legacy-compat">
<html manifest="pamflet.manifest">
      <head>
        <title>MongoDB+Hadoop Connector — Combined Pages</title>
        <link type="text/css" media="screen, projection" rel="stylesheet" href="css/blueprint/screen.css"></link>
        <link type="text/css" media="screen and (min-device-width: 800px), projection" rel="stylesheet" href="css/blueprint/grid.css"></link>
        <link type="text/css" media="print" rel="stylesheet" href="css/blueprint/print.css"></link> 
        <!--[if lt IE 8]>
          <link rel="stylesheet" href="css/blueprint/ie.css" type="text/css" media="screen, projection"/>
        <![endif]-->
        <link type="text/css" media="screen, projection" rel="stylesheet" href="css/pamflet.css"></link>
        <link type="text/css" media="print" rel="stylesheet" href="css/pamflet-print.css"></link>
        <link type="text/css" media="screen and (min-device-width: 800px), projection" rel="stylesheet" href="css/pamflet-grid.css"></link>
        
        <script src="js/jquery-1.6.2.min.js"></script>
        <script src="js/jquery.collapse.js"></script>
        <script src="js/pamflet.js"></script>
        <script type="text/javascript" src="js/prettify/prettify.js"></script><link type="text/css" rel="stylesheet" href="css/prettify.css"></link><script type="text/javascript"><!--
        window.onload=function() { prettyPrint(); };
      --></script>
        <meta charset="utf-8"></meta>
        <meta name="viewport" content="width=device-width, initial-scale=1"></meta>
      </head>
      <body>
        <a class="page prev nav" href="Contents+in+Depth.html">
            <span class="space">&nbsp;</span>
            <span class="flip">❧</span>
          </a>
        <div class="container">
          <div class="span-16 prepend-1 append-1">
            <div class="top nav span-16 title">
              <span>MongoDB+Hadoop Connector</span> — Combined Pages
            </div>
          </div>
          <div class="span-16 prepend-1 append-1 contents">
            <div class="tocwrapper show">
      <a class="tochead nav" style="display: none" href="#toc">❦</a>
      <a name="toc"></a>
      <h4 class="toctitle">Contents</h4>
      <div class="tocbody">
      <div><a href="#MongoDB%2BHadoop+Connector">MongoDB+Hadoop Connector</a></div><ol class="toc"> <li><div><a href="#Frequently+Asked+Questions">Frequently Asked Questions</a></div></li><li><div><a href="#Getting+Started">Getting Started</a></div><ol class="toc"> <li><div><a href="#Building+the+Adapter">Building the Adapter</a></div></li><li><div><a href="#Configuration+%26+Behavior">Configuration &amp; Behavior</a></div></li> </ol></li><li><div><a href="#Hadoop+Streaming+Support">Hadoop Streaming Support</a></div><ol class="toc"> <li><div><a href="#Building+Hadoop+Streaming+Support">Building Hadoop Streaming Support</a></div></li> </ol></li> </ol></div></div><h1 id="MongoDB%2BHadoop+Connector">MongoDB+Hadoop Connector</h1><p><strong>CURRENT RELEASE</strong>: 1.0.0-rc1
</p><p>The <em>Mongo+Hadoop Connector</em> (for brevitys sake, we’ll often refer to it as <em>mongo-hadoop</em> in this documentation) is a series of plugins for the <a title="Apache Hadoop" href="http://apache.hadoop.org">Apache Hadoop Platform</a> to allow connectivity to <a title="MongoDB" href="http://mongodb.org">MongoDB</a>. This connectivity takes the form of allowing both reading MongoDB data into Hadoop (for use in MapReduce jobs as well as other components of the Hadoop ecosystem), as well as writing the results of Hadoop jobs out to MongoDB. A forthcoming release will also allow for reading and writing static BSON files (ala <em>mongodump / mongorestore</em>) to allow offline batching; commonly, users find this to be a beneficial feature to run analytics against backup data.
</p><p>At this time, we support the “core” Hadoop APIs (now known as <a title="Hadoop Common" href="http://hadoop.apache.org/common/">Hadoop Common</a>), in the form of <em>mongo-hadoop-core</em>. There is additionally support for other pieces of the Hadoop Ecosystem, including <a title="Apache Pig" href="http://pig.apache.org">Pig</a> for ETL and <a title="Hadoop Streaming" href="http://hadoop.apache.org/common/docs/current/streaming.html">Streaming</a> for running Mongo+Hadoop jobs with Python (future releases will support additional scripting languages such as Ruby). Although it is not dependent upon Hadoop, we also provide a connector for the <a title="Flume" href="https://github.com/cloudera/flume/wiki">Flume</a> distributed logging system.
</p><h2 id="Support">Support</h2><p><em>mongo-hadoop</em> is dependent upon the MongoDB Java Driver — currently version 2.7.3.
</p><p>Bugs &amp; Features should be tracked and requested on the <a title="MongoDB Jira" href="https://jira.mongodb.org/browse/HADOOP/">MongoDB Jira</a>. If you have questions please email the
<a title="MongoDB User Mailing List" href="http://groups.google.com/group/mongodb-user">mongodb-user Mailing List</a>,
rather than directly contacting contributors or maintainers.
</p><h3 id="Maintainers">Maintainers</h3><ul><li>Brendan McAdams <brendan@10gen.com>
</li><li>Jeff Yemin <jeff.yemin@10gen.com>
</li></ul><h4 id="Contributors">Contributors</h4><ul><li>Eliot Horowitz  <erh@10gen.com>
</li><li>Ryan Nitz 
</li><li>Joseph Shraibman <jks@iname.com> (Sharded Input Splits)
</li><li>Sumin Xia <xiasumin1984@gmail.com> (Sharded Input Splits)
</li><li>Priya Manda <priyakanth024@gmail.com> (Test Harness Code)
</li><li>Rushin Shah <rushin10@gmail.com> (Test Harness Code)
</li><li>Sarthak Dudhara <sarthak.83@gmail.com> (BSONWritable comparable interface)
</li></ul><h1 id="Frequently+Asked+Questions">Frequently Asked Questions</h1><h3 id="Do+the+MongoInputFormat%2FMongoOutputFormats+use+HDFS%3F">Do the MongoInputFormat/MongoOutputFormats use HDFS?</h3><p>No. The <code>Mongo\*Format</code> code is designed to not use HDFS, instead reading and writing data directly between MongoDB + Hadoop.
</p><p>A forthcoming release will offer a <code>BSONInputFormat</code> and <code>BSONOutputFormat</code> which will allow for working offline with MongoDB backup files (in BSON format) on HDFS and S3.
</p><h3 id="How+does+the+MongoDB+%2B+Hadoop+Connector+differ+from+Sqoop%3F">How does the MongoDB + Hadoop Connector differ from Sqoop?</h3><p>From the <a  href="https://github.com/cloudera/sqoop/wiki">Sqoop Wiki</a>: <em>“Sqoop is a tool designed to import data from relational databases into Hadoop. Sqoop uses JDBC to connect to a database … and automatically generates the necessary classes to import data into the Hadoop Distributed File System (HDFS)“</em>
</p><p>The MongoDB + Hadoop Connector does not work with HDFS, instead reading and writing directly between MongoDB and Hadoop for the highest possible performance.  This also allows for Hadoop jobs to have the freshest possible view of their input data without an intermediary export process.
</p><h4 id="Is+integration+possible+between+MongoDB+and+Sqoop%3F">Is integration possible between MongoDB and Sqoop?</h4><p>As MongoDB is neither a relational database nor utilizes JDBC for connectivity, integration with Sqoop does not seem feasible at this time.
</p><p>A forthcoming release of the MongoDB + Hadoop Connector will offer a <code>BSONInputFormat</code> and <code>BSONOutputFormat</code> which will allow for working offline with MongoDB backup files (in BSON format) on HDFS and S3, without a live MongoDB database.
</p><h1 id="Getting+Started">Getting Started</h1><p>To get started with MongoDB + Hadoop, you’ll need a few things:
</p><ul><li>A MongoDB Installation (<em>mongo-hadoop</em> supports all MongoDB configurations including Sharding)
</li><li>A working Hadoop Installation (a list of supported Hadoop distributions &amp; versions is provided later in this document)
</li><li>A basic understanding of how to work with Hadoop and Java projects.
</li></ul><h1 id="Building+the+Adapter">Building the Adapter</h1><p>The Mongo-Hadoop adapter uses the
<a title="SBT Build Tool" href="https://github.com/harrah/xsbt">SBT Build Tool</a> tool for
compilation. SBT provides superior support for discreet configurations
targeting multiple Hadoop versions. The distribution includes
self-bootstrapping copy of SBT in the distribution as <code>sbt</code>.  Create a
copy of the jar files using the following command:
</p><pre><code>./sbt package
</code></pre><p>The MongoDB Hadoop Adapter supports a number of Hadoop releases. You
can change the Hadoop version supported by the build by modifying the
value of <code>hadoopRelease</code> in the <code>build.sbt</code> file. For instance, set
this value to:
</p><pre><code>hadoopRelease in ThisBuild := &quot;cdh3&quot;
</code></pre><p>Configures a build against Cloudera CDH3u3, while:
</p><pre><code>hadoopRelease in ThisBuild := &quot;1.0&quot;
</code></pre><p>Configures a build against Hadoop 1.0 from the mainline Apache distribution.
</p><p>We also publish releases to the central Maven
repository with artifacts customized using the dependent release
name. Our “default” build has no artifact name attached and supports
Hadoop 1.0.
</p><p>After building, you will need to place the <em>mongo-hadoop-core</em> jar and the
<em>mongo-java-driver</em> in the <code>lib</code> directory of each Hadoop server.
</p><h5 id="Streaming+Support">Streaming Support</h5><p>Certain builds of Hadoop do not support features necessary for <em>mongo-hadoop</em> Streaming, and building with those versions will not produce an artifact for <em>mongo-hadoop-streaming</em>.
</p><p>The MongoDB-Hadoop Adapter supports the following releases. Valid keys
for configuration and Maven artifacts appear below each release.
</p><h4 id="Apache+Hadoop+1.0.0">Apache Hadoop 1.0.0</h4><p>This includes Pig 0.9.2 and does <em>NOT</em> support Hadoop Streaming.
</p><ul><li>1.0
</li><li>1.0.x
</li><li>Maven artifact: “org.mongodb” / “mongo-hadoop_1.0.0”
</li></ul><h4 id="Apache+Hadoop+0.20.205.0">Apache Hadoop 0.20.205.0</h4><p>This includes Pig 0.9.2 and does <em>NOT</em> support Hadoop Streaming.
</p><ul><li>0.20
</li><li>0.20.x
</li><li>Maven artifact: “org.mongodb” / “mongo-hadoop_0.20.205.0”
</li></ul><h4 id="Cloudera+Release+3">Cloudera Release 3</h4><p>This derives from Apache Hadoop 0.20.2, but includes many custom
patches. Patches include binary streaming, and Pig 0.8.1.  This
target compiles <em>ALL</em> Modules, including Streaming.
</p><ul><li>cdh3
</li><li>Maven artifact: “org.mongodb” / “mongo-hadoop_cdh3u3”
</li></ul><h4 id="Apache+Hadoop+0.23">Apache Hadoop 0.23</h4><p>(Currently building against 0.23.1)
</p><p>This is an alpha branch of Hadoop; despite the misleading version numbers, Apache Hadoop 0.23 is “newer” than Apache Hadoop 1.0. Hadoop 0.23 is also the basis for Cloudera’s CDH4 Beta. This target compiles <em>ALL</em> modules, including Streaming and Pig 0.9.2.  Note however that we <em>do not</em> support the next-generation <a  href="http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a> at this time; support is planned for <em>mongo-hadoop</em> v1.1.
</p><ul><li>0.23
</li><li>0.23.x
</li><li>Maven Artifact: “org.mongodb” / “mongo-hadoop_0.23.1” 
</li></ul><h4 id="Cloudera+Release+4+%28Beta+1%29">Cloudera Release 4 (Beta 1)</h4><p>This is the latest beta of Cloudera’s distribution, based upon the 0.23 alpha branch of Hadoop; despite the misleading version numbers, Apache Hadoop 0.23 is “newer” than Apache Hadoop 1.0. This target compiles <em>ALL</em> modules, including Streaming and Pig 0.9.2.Note however that we <em>do not</em> support the next-generation <a  href="http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/YARN.html">YARN</a> at this time; support is planned for <em>mongo-hadoop</em> v1.1.
</p><ul><li>cdh4
</li><li>Maven Artifact: “org.mongodb” / “mongo-hadoop_cdh4b1” 
</li></ul><h3 id="Apache+Hadoop+0.22.0">Apache Hadoop 0.22.0</h3><p>This includes Pig 0.9.1 and Hadoop Streaming.
</p><ul><li>0.22
</li><li>0.22.0
</li></ul><h4 id="Apache+Hadoop+0.21.0">Apache Hadoop 0.21.0</h4><p>This includes Pig 0.9.1 and Hadoop Streaming.
</p><ul><li>0.21
</li><li>0.21.x
</li></ul><p>This build is <strong>not</strong> published to Maven because of upstream
dependency availability. 
</p><p>Unfortunately, we are not aware of any Maven repositories that contain
artifacts for Hadoop 0.21 at present. You may need to resolve these
dependencies by hand if you chose to build using this
configuration. 
</p><h1 id="Configuration+%26+Behavior">Configuration &amp; Behavior</h1><h2 id="Hadoop+MapReduce">Hadoop MapReduce</h2><p>This package provides working <em>Input</em> and <em>Output</em> adapters for MongoDB.  You may
configure these adapters with XML or programatically. See the
WordCount examples for demonstrations of both approaches.  You can
specify a query, fields and sort specs in the XML config as JSON or
programatically as a DBObject.
</p><h3 id="Splitting+up+MongoDB+Source+Data+for+the+InputFormat">Splitting up MongoDB Source Data for the InputFormat</h3><p>The MongoDB Hadoop Adapter makes it possible to create multiple
<em>InputSplits</em> on source data originating from MongoDB to
optimize/paralellize input processing for Mappers.
</p><p>If <code>mongo.input.split.create_input_splits</code> is <strong>false</strong> (it defaults
to <strong>true</strong>) then MongoHadoop will use <strong>no</strong> splits. Hadoop will
treat the entire collection in a single, giant, <em>Input</em>.  This is
primarily intended for debugging purposes.
</p><p>When true, as by default, the following possible behaviors exist:
</p><ol><li><p>For unsharded the source collections, MongoHadoop follows the
</p><p> “unsharded split” path. (See below.)
</p></li><li><p>For sharded source collections:
</p><ul><li>If <code>mongo.input.split.read_shard_chunks</code> is <strong>true</strong> (defaults <strong>true</strong>) then we pull the chunk specs from the
configuration server, and turn each shard chunk into an <em>Input Split</em>.  Basically, this means the mongodb sharding system does 99% of the preconfig work for us and is a good thing‚Ñ¢
</li></ul><ul><li>If <code>read_shard_chunks</code> is <strong>false</strong> and <code>mongo.input.split.read_from_shards</code> is <strong>true</strong> (it defaults to <strong>false</strong>) then we connect to the <code>mongod</code> or replica set for each shard individually and each shard becomes an input split. The entire content of the collection on the shard is one split. Only use this configuration in rare situations.
</li></ul><ul><li>If <code>read_shard_chunks</code> is <strong>true</strong> and <code>mongo.input.split.read_from_shards</code> is <strong>true</strong> (it defaults to <strong>false</strong>) MongoHadoop reads the chunk boundaries from the config server but then reads data directly from the shards without using the <code>mongos</code>.  While this may seem like a good idea, it can cause erratic behavior if MongoDB balances chunks during a Hadoop job. This is not a recommended configuration for write heavy applications but may provide effective parallelism in read heavy apps.
</li></ul><ul><li>If both <code>create_input_splits</code> and <code>read_from_shards</code> are <strong>false</strong> disabled then we pretend there is no sharding and use the “unsharded split” path. When <code>read_shard_chunks</code> is <strong>false</strong> MongoHadoop reads everything through mongos as a single split.
</li></ul></li></ol><h3 id="%E2%80%9CUnsharded+Splits%E2%80%9C">“Unsharded Splits“</h3><p>“Unsharded Splits” refers to the system that MongoHadoop uses to
calculate new splits. You may use “Unsharded splits” with sharded
MongoDB options.
</p><p>This is only used:
</p><ul><li>for unsharded collections when
<code>mongo.input.split.create_input_splits</code> is <strong>true</strong>.
</li><li>for sharded collections when
<code>mongo.input.split.create_input_splits</code> is <strong>true</strong> <em>and</em>
<code>read_shard_chunks</code> is <strong>false</strong>.
</li></ul><p>In these cases, MongoHadoop generates multiple InputSplits. Users
have control over two factors in this system.
</p><ul><li><code>mongo.input.split_size</code> - Controls the maximum number of megabytes
of each split.  The current default is 8, based on assumptions
prior experience with Hadoop. MongoDB’s default of 64 megabytes
may be a bit too large for most deployments.
</li><li><code>mongo.input.split.split_key_pattern</code> - Is a MongoDB key pattern
that follows <a  href="http://www.mongodb.org/display/DOCS/Sharding+Introduction#ShardingIntroduction-ShardKeys">the same rules as shard key selection</a>.
This key pattern has some requires, (i.e. must have an index,
unique, and present in all documents.) MongoHadoop uses this key to
determine split points. It defaults to <code>{ _id: 1 }</code> but you may find
that its more ideal  to optimize the mapper distribution by
configuring this value.
</li></ul><p>For all three paths, you may specify a custom query filter for the
input data. <em>mongo.input.query</em> represents a JSON document containing
a MongoDB query.  This will be properly combined with the index
filtering on input splits allowing you to MapReduce a subset of your
data but still get efficient splitting.
</p><h1 id="Hadoop+Streaming+Support">Hadoop Streaming Support</h1><p><a title="Hadoop Streaming" href="http://hadoop.apache.org/common/docs/current/streaming.html">Hadoop Streaming</a> is an add-on module to Hadoop which provides a facility for composing Hadoop jobs in non-JVM languages. With streaming, any programming language (or tool) which supports access to <strong>STDIN</strong> and <strong>STDOUT</strong> can be leveraged for data processing. To that end, an adapter for using MongoDB with Hadoop Streaming is provided in this package (as long as the build of Hadoop you are using supports the Streaming features we require).
</p><p>As working with MongoDB requires the ability to manipulate <a  href="http://bsonspec.org">BSON</a>, the options for languages to use with <em>mongo-hadoop-streaming</em> are somewhat more limited.  At this time, we provide a support module to work with Python, and supporting additional languages (such as Ruby) is planned for a future release.
</p><h4 id="Availability+of+MongoDB+Streaming+Support">Availability of MongoDB Streaming Support</h4><p>Due to the manner Hadoop Streaming is used, <em>mongo-hadoop-streaming</em> requires the construction of a “fat” (sometimes known as a “shaded” or “assembly”) jar, which contains all of its dependencies including the Java Driver.  This limits our ability to publish Streaming support to Maven.  Instead, if you’d like to leverage <em>mongo-hadoop-streaming</em> you should build it yourself, or download a preassembled jar from our <a  href="https://github.com/mongodb/mongo-hadoop">Github Page</a>.
</p><h1 id="Building+Hadoop+Streaming+Support">Building Hadoop Streaming Support</h1><p>As noted in the “Building the Adapter” guide, not every version of Hadoop supports the features required to use MongoDB with Streaming. Please consult those build instructions to ensure your version of Hadoop is supported for MongoDB + Streaming.
</p><p>Due to the manner Hadoop Streaming is used, <em>mongo-hadoop-streaming</em> requires the construction of a “fat” (sometimes known as a “shaded” or “assembly”) jar, which contains all of its dependencies including the Java Driver. To this end, simply using the <code>package</code> task in <em>sbt</em> is not enough. To build a proper package, the following <em>sbt</em> command is used:
</p><pre><code>./sbt mongo-hadoop-streaming/assembly
</code></pre><p>This will create a new “fat” jar in: 
</p><pre><code>streaming/target/mongo-hadoop-streaming-assembly-1.0.0-rc1.jar
</code></pre><p>This jar file is runnable with <code>hadoop jar</code>, and contains all of the dependencies necessary to run the job. 
</p><h3 id="Setting+up+Language+Support">Setting up Language Support</h3><p>Each individual scripting language will have different requirements for working with MongoDB + Hadoop Streaming.  Once you have the jar file built for <em>mongo-hadoop-streaming</em>, you will need to build and deploy the support libraries for your chosen language.
</p><p>It will also be necessary to ensure these libraries are available on each Hadoop node in your cluster, along with the <em>mongo-hadoop-core</em> driver as outlined in the main setup instructions.  However, you do not need to distribute the <em>mongo-hadoop-streaming</em> jar anywhere.
</p><h4 id="Python+Streaming+Setup">Python Streaming Setup</h4><p>Working with Python streaming support will require two Python modules installed on each cluster node:
</p><ul><li>pymongo 2.0+ 
</li><li>PyMongo_Hadoop
</li></ul><p><em>pymongo</em> is available via the normal installation/distribution methods for Python modules.  PyMongo_Hadoop is distributed with this source, and you can build and distribute it from there.
</p><h5 id="Building+PyMongo_Hadoop">Building PyMongo_Hadoop</h5><p>You will find the source for the <em>pymongo_hadoop</em> Python module under <code>streaming/language_support/python</code>. To create a package to distribute to each of your Hadoop nodes, the easiest process is:
</p><pre><code>python setup.py bdist_egg
</code></pre><p>After which you will find an egg file underneath the <code>dist</code> directory.  You may then install this egg file to each of your Hadoop nodes.  If you are working on a single development node, you may find it simpler to run the <code>develop</code> task on setup.py rather than installing an egg file.
</p><p>The following Python script may prove useful in validating your installation on each node:
</p><pre><code class="prettyprint lang-python">#!/usr/bin/env python

try:
    import pymongo
    from bson import _elements_to_dict, InvalidBSON
except:
    raise Exception(&quot;Cannot find a valid pymongo installation.&quot;)

try:
    from pymongo_hadoop import BSONInput
except:
    raise Exception(&quot;Cannot find a valid pymongo_hadoop installation&quot;)

print &quot;*** Everything looks OK. All required modules were found.&quot;

</code></pre>
          </div>
        </div>
        <a class="fork nav" href="http://github.com/mongodb/mongo-hadoop"><img alt="Fork me on GitHub" src="img/fork.png"></img></a>
      </body>
    </html>